Add perf experiment pipeline and fix v2 connection pool regressions - #4543

Adds a third perf pipeline that A/B tests one runner-config switch against
itself on the same source build, alongside the existing package-baseline and
PR-baseline pipelines.

The run scripts gain a general --switch-under-test / -SwitchUnderTest option
(UseConnectionPoolV2, UseOptimizedAsyncBehaviour, UseManagedSniOnWindows) that
writes two runner configs differing only in that key and hands one to each
pass. These are AppContext switches latched process-wide, so they cannot be
toggled between benchmarks in a single process. Both passes share one build,
and the option is rejected alongside a source baseline so the delta stays
attributable to one variable.

The new pipeline is a separate file rather than a flag on the other two so
that skipping Kusto is structural: both rows would share a DerivedRunId,
PerfRun.Config is stamped once per run, and nothing marks a row as an
experiment.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…pletionSource

The first pass at the fast path completed the caller's TaskCompletionSource
and returned false. That moved the thread pool dispatch rather than removing
it: returning false sends SqlConnection.InternalOpenAsync down its
asynchronous completion branch, which allocates an OpenAsyncRetry, a Tuple
and a CancellationTokenRegistration, then schedules the continuation with
ContinueWith(..., TaskScheduler.Default). That continuation costs a thread
pool hop even though the result is already available.

A re-run of the experiment pipeline showed the residual cost: OpenAsyncConnection
was still +145% and RapidFireOpenClose still +23-58% with allocations up
70-163%, all of them async opens against an unsaturated pool that were hitting
the fast path and paying for the handoff anyway.

Return true with the connection instead, which is what
WaitHandleDbConnectionPool does on its own inline hit, and leave the
TaskCompletionSource untouched for the caller to abandon. InternalOpenAsync
then takes its synchronous branch. Exceptions now propagate synchronously,
which also matches v1; InternalOpenAsync already converts them into a faulted
task.

Update StressTestAsync, which awaited the TaskCompletionSource unconditionally
and so hung once requests began completing inline. It now checks the completed
flag first, matching the pattern already used in the pool transaction tests
and by the pool's real callers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A switch experiment flips intended behaviour, so benchmarks that measure that
behaviour regress by design. UseConnectionPoolV2 is the motivating case:
ChannelDbConnectionPool opens physical connections concurrently, where
WaitHandleDbConnectionPool serialises growth behind a Semaphore(1, 1), and
ConnectionPoolStressRunner.RapidFireOpenClose measures a cold-start burst where
the extra parallel opens have nothing to amortise against.

With nowhere to record that, the same benchmarks get re-investigated every run
and --fail-on-regression is unusable for experiments.

Add an optional per-switch annotation file at
expected-differences/<SwitchName>.json, picked up automatically by
--switch-under-test. Matching entries are reported as expected differences,
grouped under their reason in comparison.md, excluded from the confirmed
regression count and ignored by the regression gate.

Two limits keep the annotation honest: a reason is required, and only
regressions are reclassified, so an annotated benchmark that comes back
unchanged or improved keeps its real status.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Sync callers on ChannelDbConnectionPool waited for an idle connection by
driving Channel.ReadAsync from a blocked thread, so the wait could only be
released by a thread pool continuation. Blocked sync callers occupy the very
threads that continuation needs, so a static SemaphoreSlim sized
ProcessorCount/2 capped how many sync callers could wait at once. That guard
is process-wide and shared across every pool.

IdleConnectionChannel now gates reads on a counting SemaphoreSlim, released
directly by the writing thread. Sync callers block on SemaphoreSlim.Wait,
which needs no continuation, so the starvation risk and the throughput cap
both go away. Async callers gate on the same semaphore to keep the counts
aligned, and channel completion is surfaced through a linked token so waiters
still see ChannelClosedException.

ConnectionPoolRampRunner covers what RapidFireOpenClose cannot. That benchmark
starts cold but releases each connection immediately, so one physical
connection satisfies every caller and slow pool growth wins. The new runner
keeps the cold pool and holds each connection until all callers have one, so
the pool must open N connections and concurrent creation is what is measured.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The semaphore-gated wait serves waiters in roughly arrival order but does not
guarantee it, so the docs should not promise first-come, first-served.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Measurement does not support it. On a saturated pool (50 sync workers,
MaxPoolSize 10, so 40 concurrent waiters against 5 permits), raising the
static semaphore from 5 to 100 permits changes nothing: 302 ms vs 280 ms at
5 ms hold, 60 ms vs 59 ms at 1 ms hold. The permit is released as soon as a
connection arrives, so it caps how many callers are blocked, not throughput.

Replacing the wait with a semaphore-gated channel measured identically to the
original in every regime tested (saturated and unsaturated, 0/1/5 ms hold,
thread pool and dedicated threads). It bought nothing and cost the FIFO
fairness the pool documents as a headline property, since SemaphoreSlim does
not guarantee release ordering and acquisition became two-stage.

The residual v2 sync penalty is a fixed per-handoff cost on the wait path,
visible only when the pool is saturated: +67% to +167% with zero work per
operation, falling to +5% to +9% once each operation does 1 ms of work, and
zero when Parallelism <= MaxPoolSize. That is a different and much smaller
problem than the one this change addressed.

Keeps ConnectionPoolRampRunner, which is unrelated.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A sync Open() that waits blocks its thread. On threadpool threads that
competes with the threadpool itself, so a pool whose waiter wake-up needs a
queued continuation stalls until thread injection runs. The existing
benchmarks could not separate that from an intrinsic pool cost.

Adds a dedicated-thread variant of SteadyStateOpenQueryClose alongside the
threadpool one, and a ConnectionPoolThreadPoolPressureRunner that pins the
threadpool floor above and below the worker count so the effect is
reproducible rather than dependent on hill-climbing timing.

Existing benchmarks and their params are unchanged, to keep trend continuity.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

SteadyStateOpenQueryClose runs 50 synchronous workers on thread pool threads
against a pool of 10, so most of them block in Open() and the thread pool has
no free worker left to run the waiter wake-up. That is thread pool saturation,
not pool throughput: medians match within 2% and the whole delta is tail
latency, which disappears when the thread pool floor is raised.

Keeping parallelism below the thread pool's worker count is the application's
responsibility, as is pre-warming the thread pool, so this is annotated rather
than fixed. ConnectionPoolThreadPoolPressureRunner characterises the boundary.

Enabling AllowSynchronousContinuations on the idle channel was considered and
rejected: it would resume the waiter's retrieval loop on the returning thread,
and that loop can reach OpenNewInternalConnection, which opens a physical
connection synchronously. A caller's Close() could then pay for another
caller's connect, TLS handshake and login.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Measured against the in-process TDS server. A physical connection costs ~57 KB
and both pools allocate the same per connection (v1 57,275 B, v2 57,701 B).
Across all six RapidFireOpenClose parameter sets, the allocation delta divided
by the extra Login7 count lands at 55.6-56.2 KB, so connection count accounts
for the whole delta with nothing left for per-operation overhead.

On a warm pool where neither implementation opens anything, v2's checkout and
return path allocates 24-48 B/op more than v1 (+0.2% to +0.5%).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Switch experiments surface intended behaviour changes as regressions. The
annotation file was one way to record that verdict; explaining it in review
is another, and it avoids a mute mechanism that has to be trusted.

Removes the per-switch JSON rules, their loader and matcher, the
expected-difference status and report section, and the --expected-differences
plumbing through both run-perf-tests wrappers.

The README guidance on writing benchmarks that measure the intended behaviour
directly (ConnectionPoolRampRunner, the dedicated-thread and threadpool-pressure
variants) is kept, since it stands on its own.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…dance

The section still read as if 'intended differences' were a category the
pipeline recognised. It no longer is. Retitles it, states plainly that there
is no mute mechanism and why, and points at writing a benchmark that measures
the intended behaviour instead.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>